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Abstract 

The orientation of patches on the surface of an object can be determined from multiple 
images taken with different illumination, but from the same viewing position. This method, 
referred to as photometric stereo, can be implemented using table lookup based on 
numerical inversion of experimentally determined reflectance maps. Here we concentrate on 
objects with specularly reflecting surfaces, since these are of importance in industrial 
applications. Previous methods, intended for diffusely reflecting surfaces, employed point 
source illumination, which is quite unsuitable in this case. Instead, we use a distributed 
light source obtained by uneven illumination of a diffusely reflecting planar surface. 
Experimental results are shown to verify analytic expressions obtained for a method 
employing three light source distributions. 


This report describes research done at the Artificial Intelligence Laboratory of the 
Massachusetts Institute of Technology. Support for the Laboratory’s artificial intelligence 
research is provided in part by the Office of Naval Research under contract N000H-77-C- 
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Notation 




© : zenith angle (local coordinate system) 

$ : azimuth angle (local coordinate system) 

6 : zenith angle (viewer coordinate system) 

</>: azimuth angle (viewer coordinate system) 
F : flux [ watt/sr ] 

E : irradiance [ watt/m 2 ] 

L : radiance [ watt/(m 2 sr) ] 

- suffix -- 
s : source 
j: incident 
e : emitting 
p : image (or picture) 

„: surface normal 
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0 Overview 

The technique of photometric stereo has been proven by an experiment whose schema is 
shown in Fig. i. Orientations of surface patches on a sphere obtained using this technique 
are shown in Fig. 2. 
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Fig. 1 The overall schema of the experiment. The experiment has two parts: initialization, 
represented by broken lines, and the lookup of surface orientations for a given object using 
measured image brightness. Brightness arrays obtained from a TV camera are normalized. 
A lookup table is constructed in the initial process using reflectance map techniques and 
Newton’s method. The table is used to obtain surface orientations. 
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Fig. ?. Needle diagram showing (projected) surface normals on the surface of a metallic 
sphere. Each needle shown sticking up out of a point represents the orientation of a surface 
parch. 
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1 Introduction 

The geometric dependence of the reflectance characteristics of a surface can be expressed in 
terms of the direction from the surface to the viewer, the direction from the surface to the 
light-source and the surface normal. The apparent brightness (scene radiance) seen by the 
viewer can be expressed as a function of the three angles between these directions. 

Alternatively, we can express the dependence in terms of the slope components p and q, 
used as axes in gradient space [1], There are several ways to project the unit surface normal 
onto a plane. In the "traditional" method one projects a point on the unit sphere onto a 
tangent plane from a center placed at one pole of the unit sphere. The plane is tangent to 
the sphere at the opposite pole (stereo-graphic projection). This projection may be useful 
when we consider characteristics of a surface which has both specularity and transparency 
like the surface between air and water or windows of the John Hancock building; one 
hemisphere corresponds to reflected brightness distribution, the other to transmitted 
brightness distribution. 

p’= dzIdxK yV(<3z/<3x) 2 +(3z/dy) z t 1 ) ( | ) 

q'= <?z/ciy/( ,/l+(clz/<3x) 2 +(c)z/3y) 2 t 1), 

where z=Z(x,y) and the sign is chosen depending on whether the point is in the upward 
hemisphere or the downward hemisphere. In another method, which has been used by 
Horn [1], one projects the unit surface normal onto the tangent plane from the center of the 
unit sphere (central projection). 


p = Az/Ax ( 2 ) 

q = Az/Ay 



(a) The stereographic projection. (b) Horn’s projection. Note its simplicity. 

Fig. 3 Horn's projection and the stereographic projection onto a plane. 

Although the traditional method may be more familiar and allows one to express any 
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direction without ambiguity, the zenith angle is projected into half of the zenith angle and, 
thus, the coordinate transformation is not as convenient as it is for the second method. 
Moreover, we only need to consider the upper hemisphere of possible directions, since 
surfaces turned away from the viewer cannot be seen in any case. It is for these reasons 
that we have chosen the gradient space projection. 

If we take the direction from the surface to the viewer as the direction of the z-axis, then 
the reflectance properties of a surface patch depend on (p,q), the direction of the surface 
normal and (p s ,q s ), the direction to the source [1], Each point in gradient space (the tangent 
plane), corresponds to a particular surface orientation (based on the direction of the viewer). 
If we know the reflectance characteristics of an object, we can calculate how bright a surface 
element with that orientation will appear. It is convenient to use contour lines to connect 
those points in gradient space which correspond to surface orientations which give rise to 
the same apparent brightness. It is because of these contour lines that the resulting diagram 
is referred to as the "reflectance map" [I]. It is denoted by R( p,q ). 

Using the reflectance map, the basic imaging equation is 

E,(x,y) = Rj( p,q,x,y ), ( 3 ) 

where E,(x,y) is the brightness (image irradiance) in the image-forming system at the point 
(x,y) in the image plane. This equation contains two unknown variables p, q and one 
quantity E r which can be measured in the image. 

In the above equation, the subscript j is used to denote different illumination conditions. 
For each value of the subscript, a different image is obtained, and a different reflectance 
map applies. If two images are taken, two such equations provide constraints on the 
possible values of p and q. This permits us to solve for the gradient. Because of the non¬ 
linearity of the equations, however, a number of solutions may be found at times. In this 
case, a third image may be used to disambiguate the remaining possibilities. This is the 
principle of photometric stereo [2,3,1 

i 

Orthographic projection can simplify the calculation considerably. If we can assume that 
the object is small compared with the distance to the source and the image-forming system, 
then the viewer direction can be approximated as the axis of the image-forming system, and 
we can treat the system as orthographic. There are two merits to this approximation. One 
is that we can neglect the effect of position. The right hand side of Eq. 3 depends only on 
(p.q); namely, we can apply the same reflectance map on all point in the image. Another 
benefit is that we can calculate R( p,q ) more easily because the approximation fixes the 
viewing direction. So we can, for example, rotate the source keeping the phase angle 
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constant (The phase angle is the angle between the source and the viewer, measured at the 
object). This means that we can obtain a new reflectance map just by rotating the old one 
t3l 

2 Basic Consideration 

2.1 Relationship between Radiance and Irradiance 

One of the main points of our discussion here is that we consider only specular components 
of reflectance when we calculate the reflectance map, since many industrial materials are 
made of metal and have strong specularity and little diffuse reflection. Experiments show 
that only 1 or 2% of the incident light is reflected diffusely from some metallic surfaces, with 
most of the rest reflected specularly. We cannot treat this kind of material using the usual 
Lambertian model for reflection of light from a surface. It is also clearly inappropriate to 
use point sources to illuminate such a surface, since very few surface patches will be oriented 
correctly to reflect any light and we will only see virtual images of the point sources. 

We consider two kinds of light sources. One is an extended light source. The other is a 
collimated one. An example of an extended light source is a fluorescent light fixture. A 
collimated light may be generated by using a laser and an inverted telescope. 

Three relationships exist between a light source and the image plane. The first one is that 
between source radiance and incident irradiance on a sample surface. Next, and most 
important, is that between incident irradiance and emitting radiance from the surface (this 
is captured in the reflectance map). The last one is that between emitting radiance and 
image irradiance. Since the two kinds of light sources have somewhat different 
relationships, we consider them separately. 

(extended source) 

The relationship between source radiance from an extended light and the incident 
irradiance is fairly straightforward. We assume a surface patch of a light source (dA) is 
emitting energy to a surface patch of a sample surface (dB). The emitted energy from the 
source incident on the sample surface is 

dF s = L s dWidAcos©j ( 4 ) 

= L s (l/r 2 )dAdBcos© s cos©j. 

where w; is a solid angle corresponding to the surface patch dB viewed from dA, and r is 
the distance between the source and the sample surface. On the other hand, the received 
flux is 
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Fig. 4 Relationships between light source, surface, and image plane are depicted. 


dFj = L,dw s dBcos© s ( 5 ) 

= Li(l/r 2 )dAdBcos©jC.os® s , 

where o) s is a solid angle corresponding to the surface dA of the source viewed from dB. 

If L s and L; can be regarded as constant within the small areas dA, dB, then 

Lj = L s . < 6 > 

This implies that we can regard the "incident radiance" as the source radiance in the same 

direction. 

For a specular surface and an extended source, we can obtain from the definition of the bi¬ 
directional reflectance distribution function (BRDF) [1,4] 

L.<©..*.) = //f eii dEj 

= //26(sin 2 0 B -sin 2 ©i) §(4 > B -'fy 7r ) L l (© i ,4 , j)cos0 j sin© l d© i d4>, 


= L i (© B ,‘V 7r ), 

(7) 

where 


f i = 26(sin 2 © B -sin 2 © i )S($ B -4T 7r ) 

(8) 


is the BRDF for a perfectly specular surface; Li(© e ,4> e+ 7r) is the "incident radiance", and 
L 0 (© 0 , < 1'J is the reflected radiance. This result is also predicted from elementary physical 

V v 

laws. Thus, even though the source distribution may be complicated, only the contribution 
from a single direction (©.JyTr) need be considered and one need not be concerned with 
the effects of other areas of the distributed light source. Strictly speaking, Lj is not defined 
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as a function of local angles but viewer angles. Light source which we can construct has a 
fixed brightness distribution on the viewer system. However, it is easier to understand the 
result expressed using the above notation. In the next section we will give the equations for 
the transformation between the viewer coordinate system and the local coordinate system 

The relationship between reflected (scene) radiance and image irradiance is 

E p « {(7r/4)(d/f p ) 2 cos 4 or}L e , (9) 

where f p , d, a are effective focal length of the lens, the diameter of the entrance aperture 
and the off-axis angle, respectivelyt 2 3. 

l itially, using on Eq. 6,7,9 we see that image irradiance at a particular point is proportional 
to the source radiance in a direction which depends on the orientation of the corresponding 
surface patch! 1 ]. That is, the brightness of a particular surface patch is simply equal to the 
brightness of the part of the extended source which it happens to reflect. Thus, even 
though the source distribution may be complicated, only the contribution from a single 
direction (p s ,q s ) need be considered at any time. 

(collimated light source) 

In the case of a collimated source, it is easy to use irradiance to measure relationship 
between source intensity and incident brightness. Clearly, provided that a surface patch is 
small compared to the diameter of the source, source irradiance E s is the same as E, 
measured on a surface lying orthogonal to the rays. According to [ I 3, incident radiance L, 
is 

L,(©i,^) = Ei6(©j-© s )S($j-$ 8 )/sin© s . ( 10 ) 

The beauty of this expression is that the double-delta expression has dimension 
[ l/steradian 3, because when it is integrated with respect to solid angle the result has no 
dimension. So the total dimension of the right hand side of Eq. 10 is [ watt/(m z sr) ] and it 
.corresponds to the left hand side. If all the direction are covered with collimated sources 

each of which has a particular brightness distinguished with the direction, 

= .tkE^Gj -©s^r^sinQi, (II) 

where k denotes the k-th light source. 

Now we can calculate the scene radiance by using the BRDFf 1,4 ]. The BRDF for the 

surface is the same as in the former case and is given in Eq. 8. 

LpfQg.'I’p) =//Lj (© i , , I> j )26(sin 2 © e -sin z ©j)6( , F e -4> j +7r)cos©iSin©id0 j d4’ j (12) 

= L,(© e ,$ r +7r) 

Again, the same abbreviation is used; namely, brightness distribution E k is not defined as 
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a function of the local coordinates. This will be discussed in the next section. 


The relationship of scene radiance and image irradiance is, of course, the same as for the 
extended light source. In this case we also find that, using Eqs. 11, 12, 8, the image 
irradiance can be calculated from the source irradiance determined by the orientation of the 
local normal on the surface patch. 

(transformation from a local coordinate system to a viewer coordinate system) 

It is convenient to change from the local coordinate system to the viewer-centered coordinate 
system [1] because the reflectance map is defined in a viewer-centered coordinate system, and 
the light source distribution is given based on the direction of the viewer. 


In both cases, we have to determine the viewer coordinates corresponding to (© e .4v7r) 
when the viewer and the surface normal have the relationship specified by (© e ,4 e ). By 
using the relations 

©i = © G = 0 n , 

ij = 4> e + 7T, ( 13 ) 

we get the corresponding directions of the viewer coordinate 

f?, = 20 n 

4>i = 4> n - ( 14 ) 

This relationship is obtained either by considering Fig. 5 or substituting Eq. 13 into 

spherical trigonometric relations. 


Thus, we can rewrite Eq. 7 and Eq. 12 using the viewer-center coordinate system. 

Ep^rv^n) = aL(0 n ,<f> n ) = otL(20 n ,$ n ), 
where « is a constant. We observe this E p as image irradiance. 

From Eq. 14 we can also obtain 
Ps = ^ p n / (l'pn '^n ) 

q s - 2 q n / (hVV). (15-D 

We can finally express brightness distribution in the gradient space. In the case of an 

l 

extended light source, there is no problem. 

R(pn.<]n) = L s (p s .q s ), ( 161 ) 

where 

p n = 0z/c)x = -cos$ n tan0„ 
q n = dzldy = -sin<£ n tan0 B . 


ystem and a viewer coordinate system. 


k 

$ 


and (f) k « <£ n . We finally get 

( 16,2 ) 
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2.2 Consideration on Types of Brightness Distribution. 

The mapping (p n ,q n )eU = {(x,y)|(x 2 +y 2 )<l} to <p s .q s >e v =R 2 -{oo} has the following 
characteristics. 

0) (pn.q n )=o -> (ps-q s )=o 

(2) p n =0 -> p s =0 

(?) q n =0 -> q s =0 

(4) p„/qn-p 8 /qs are non-zero) 

(5) p 2 +q 2 =(/T+p7^q7 - 0 2 /(p s 2+ q s 2 ) 

Actually,the forward transformation is given in Eq. 15.1. Namely, 

ps = 2 p J <l-p 2 -q 2 ) 

q s = 2 q n / 0-p 2 -q 2 ). ( 15 1 ) 

On the other hand, the inverse transformation is 

Pn = 

qn = q s ( Ap s 2 + q s 2 * »)/(p s 2 -q s 2 ) < 15 2 > 

We will use a differentiable single-valued function as the reflectance function. Roughly 
speaking, since we determine surface orientation from brightness, if the function is not 
single valued, the inverse function theorem does not apply, and this makes the situation 

very difficult. 

From the consideration given in the Appendix, the reflectance map is a regular surface 
based on the choice of the reflectance function as a differentiable single-valued function. It 
means that at each point on the surface each point possesses a neighborhood where we can 
define an infinite derivative on the surface independent of choices of parameterization. It 
also turns out that (p„,q n ) and (p s ,q s ) are two parameterizations of the surface. This means 

that each point on the surface uniquely corresponds to a particular point (p n ,q n ) and Cp s ,q s ) 

* 

and that (p n ,q n ) and (p s ,q s ) are representations of the point by using two different coordinate 
systems. We can treat the surface using one or the other coordinate at our convenience and 
we can always translate from one to the other without ambiguity. These properties will be 
used when we build the lookup table using Newton’s method. 

2.3 Shape of Light Source and its Brightness Distribution 

Basically, we considered two shape for making a light source under the condition of 
differentiable one-to-one mapping. One is a sphere and the other is a plane. Table 1 
contains their shapes and brightness distributions. 



Table 1 


3 Application Problems 

3.1 Total Schema of the System 

The technique requires two kinds of tasks; one is the off-line (pre-computing) job and the 

othei is the on-line (real-time) job which is rather simple compared with the off-line job. 

The simplicity implies rapid calculation, as desired in any hand-eye system. The off-line 

job consists of making the reflectance map and constructing a lookup table. The on-line job 

consists of reading the image brightness and determining orientations of a surface patch 

based on the lookup table. Fig. 6 shows the information flow between the on-line job and 
the off-line job. 
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Fig. 6 The overall schema of the experiment. The technique consists of two types of jobs: 
off-line preparation of tables (represented by broken lines) and on-line processing. Image 
brightness is obtained by using a TV camera. We search for the maximum value of 
brightness in each array. Brightness arrays are calibrated and normalized. Surface 

orientations are obtained from the lookup table. 

3.2 Consideration of Light Source 

We use a Lambertian surface as a source plane. It is illuminated by a linear lamp as shown 
in Fig. 7. Though a spherical shaped source can easily cover directions of more than ninety 
degrees, it is difficult to build such a device and difficult to control the distribution of the 
light on it, particularly if interflection are taken into account. On the other hand, if we use 
a planar source illuminated by a lamp, the brightness distribution is complicated, but we 
can calculate a reflectance map as in case 2 of Table 1. It is possible to cover angles of more 
than ninety degrees by making a box-like source. In that case, however, we have to treat 
each plane separately because the surface normal is not differentiable at the intersection of 
two planes. Thus, we consider one plane surface which is assumed to have the Lambertian 
characteristics in order to obtain some analytic results to help in the design of the system. 
This plane is illuminated by a line source as a representative case. 

Brightness distribution on the surface is calculated using Eq. 18. Let f be the flux late per 
unit source length [ watt/(m sr)]. Then the total irradiance E [ watt/m 2 } is 

E = /fcos0|cos# 2 /r 2 dt. ( 17 ) 

From cos0| = y'x z +l 0 2 /r and cos 0 Z = l 0 /r, w * can finally get 
E = (fl 0 /2a 2 )[{tan' 1 (y+L)/a - tan‘'(y-L)/a) 

+ { a(y+L)/(a 2 +(y+L) 2 )-a(y-L)/(a 2 +(y-L) 2 )}), < 18 > 
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Fig. 7 The object is illuminated solely by the light reflected of an overhead Lambertian 
surface. This surface receives light from a linear lamp positioned below so as not to 
illuminate the object. The TV camera peers through a small hole in the overhead surface. 
Brightness distribution from a linear lamp on the Lambertian surface can be calculated in 
Eq. 18. We used three linear lamps, placed symmetrically, 120 degree. In the center of the 
lamps there is a hole. The object is observed through the hole. 
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where a = ,/x 2 +l 0 2 , L is the length of the line source, and l 0 is the distance between the line 
source and the Lambertian surface. 

We put an object just under the point (x,y) = (0.5l o ,0.0). E(x,y) is a symmetric function with 
respect to y. Thus' it is natural to put an object somewhere on the x axis. E(x,0) is an s- 
shape function and has a inflection point near x = 0.5l o . At that point 
E(0.5l o ,0.0)=0.715E(0.0,0.0). So it is convenient to put an object just under this point. If we 
denote the distance from the surface to the object as t ( , 
x = -l,p s +l 0 /2 

y = iiq s <19) 

Finally, we can get a reflectance map 

R(pn.q n ) = Cn 0 /2{(-i, p s +i 0 /2) 2 +i 0 2 ]3[ _______ 

ta n" 1 {(-I ,q s -d 0 /2)/ vfl|p s d 0 /2) z +l 0 z }-tan’ 1 {(-l|q s -l 0 /2)/ Xl ip s + *o/2> z +l 0 2 } 

+ { /-i | Psdo/2) z ->l 0 z (-| |q s 4| 0 /2))/ {{-I ip $ +l 0 /2) 2 +l 0 2 +(-l ,q s +l 0 /2) 2 } 
~{\/( : i|Ps + V2) 2+ l 0 2 (‘l|qs'l 0 /2)}/{(‘l l Ps + V2) 2+ l 0 2+ (‘l|qs'I 0 /2) 2 }]. < 20 > 

where p s and q g are as defined above, and L = l 0 ,2l ( = l 0 because these values are found to 
be optimal upon simulation. 

We used three linear lamps, placed symmetrically, 120° apart, about the hole through which 
the object is observed. One lamp is turned on at a time, giving rise to a reflectance map 
like the one shown in Eq. 20. 

By using three reflectance maps, we can determine the surface orientation. Theoretically it 
is possible to determine the orientation by using two maps only. However, there may be 
more than one solution because of the non-linearity of the equations. If we use three maps, 
we can determine a unique solution easily as shown in Fig. 8. 

3.3 How to Make a Lookup Table 

The most convenient method for converting a triple of measured brightness to an 
orientation is by means of a lookup table made from the reflectance maps. This table, 
indexed by quantized brightness measurements, contains surface orientation. Although it is 
possible to make a three dimensional lookup table in which each dimension corresponds to 
the brightness of a surface patch under one of the three sources, the weakest brightness 
contains relatively large measurement errors and we use it only to choose a solution between 
two alternatives. Thus, the lookup table can be two dimensional. We look up an entry 
using the two largest brightness values. Each entry contains two alternative solutions. Each 
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Fig. 8 How to use the reflectance map method. Three reflectance maps drawn in the same 


gradient space. Since the lamps are symmetrically configured, brightness distribution 
corresponding to a linear lamp is also symmetric having the same shape. Thus, we need to 
calculate only one distribution and then rotate it 120 degrees. The resulting function is the 
desired one. 

We can determine (p,q) from three values of brightness, where each brightness value 
corresponds to each source condition. For example, switching on source 1 will yield a 
brightness value of 0.7822 at cell B. Similarly, source 2 and source 3 yield 0.5829 and 0.9150, 
respectively. From the above diagram, we note that the point (-0.1 -0.4 ) satisfies this triple. 
Hence, the surface orientation at the cell R is ( -0.1 -0.4 ). 
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alternative solution also contains a surface orientation and the third brightness. 

We constiuct the lookup table by using Newton’s method. An elegant method exists in the 
case of a Lambertian reflector and point sources [2], In our case, however, we have to 
calculate it numerically. Specifically, we solve two expressions like Eq. 20 for p and q. The 
expressions differ in the brightness distribution of the source, obtained by turning on one of 
the three linear lamps. 

Theie are two choices for the calculation. One method, which we rejected, divides the 
gradient space into a mesh and gets a solution by interpolation. A similar technique is 
considered in [ 6 ] to make an exact color from a combination of three primal colors. T hey 
ttied to find the nearest grid point to a solution and then determine which neighboring box 
(the small space between grid points) contains the solution. Finally, they used an 
interpolation method from vertices of the box to determine the result. The technique would 
be useful when a reflectance map is obtained experimentally. 

The other method is to solve Eq. 20 numerically for p and q. In case that reflectance maps 
aie given in analytic form, it makes sense to use Newton’s method. Moreover, we have 
found that Newton’s method converges within four iterations with relative error smaller 
than 0.001, provided that the initial point is well chosen. Since the neighboring solutions of 
the lookup table are good initial points, the calculation requires the order of 4N 2 , where N 
is the number of mesh points, while the former method needs at least N°/2 calculations 
besides the initial calculations at each mesh. 

The N-dimensional Newton method involves solving the simultaneous linear equations; 

f i + £k. , n af i /ax K | Xk = X(( l'W it|} -x k [i) ) - 0, (21) 

where f f is the i-th element of a vector function of n dimensions and x/ i! is the i-th element 
of a solution vector of the j-th iteration. 

From the consideration of Section 2.2, we can treat the reflectance map as a function of 
(PvHs) when we get a solution using Newton’s method. Newton’s method finds an 
intersections of two surfaces. Since Eq. 20 is differentiable, the result is a regular surface 
and the iteiation converges to a point along the regular surface. From the consideration in 
Appendix and Section 2.2, this situation can be observed from either of two 
parameterizations and the expression of the result in one parameterization should 
correspond to the other exactly. Thus, we use a function parameterized by (p s ,q s ) a.s f, we 
obtain a numerical solution and then we project the solution to (p n ,q n ) space. 
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We can rewrite Eq. 21 as Eq. 23. in our case of a planar source illuminated by a linear lamp. 
The two mapping functions are 

f(p s .q s ) - R(p 8 .qs)-E Pf n 

g(p s .q s ) = R(p s cos«-q s sin«,p s sina+q s cosa)-E prn , ( 22 ) 

where « is an angle between two light sources and E in , E jrn are image brightness 
corresponding to the (n,m) element of the lookup table. We can finally get 

X [M1 = xl'kFV'F, 1 ( 2? ) 

where X = '(p s %), F « '(f,g) and (F’)' 1 is a inverse of the Jaccobian matrix. 

i 

The complete algorithm is shown in Fig. 9. At first, we try to find solution correspond mg to 
a brightness pair (0.4,0.4) from initial points (1.0,1.0) and (-1.0,1.0). Next, we proceed to get 
solutions along a line on which the first brightness is constant (0.4), using former solutions 
as new initial points until two corresponding solutions from two different directions are the 
same or we cannot get solutions within five iterations. Then, we go to the next mesh point 
along a line on which the second brightness is constant (0.4). After the entire area is 
processed, the algorithm terminates. 



® (o.^o.gy 


Fig. 9 How to get entire solutions using Newton’s method. 
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t . P 

3.3 How to Get a Surface Orientation 

Image brightness is obtained from the TV camera. To reduce the noise typical of these 
devices, we took more than one picture per light source, and arrays corresponding to the 
same light source weie averaged. The resulting three brightness arrays, one for each light 

source, were the input to the photometric stereo system. 


We seatch. each anay for its maximum brightness. If objects are convex and the visual 
angle is wide enough, the image array always contains a brightness value corresponding to 
the maximum source brightness. By normalizing the results using these maximum values, 
we can cancel the effect of varying albedo and image sensor response. Namely, we can 
identify the maximum value in an image array with the value 1.0 in the reflectance map. 
However, we cannot normalize the values of the image arrays at this stage because of the 
non-linearity of the TV camera. Normalization will be done after the value is calibrated. 
In other word, since the output of the TV camera is not linearly proportional to a 
brightness value, we have to normalize the output after we convert the output to a certain 
value proportional to image brightness. It, is interesting that the maximum point always 


contains important information and this search resembles Land’s lightness search model 
t 7 ]. 


Brightness calibration is done using a six step KODAK gray scale. Normalization is based 
on the maximum value. So each point on the image has a triple of calibrated brightness 

under three different light sources. 


From the lookup table we can get the surface orientation. The lookup table entry is 
accessed using the two largest brightness values. The lookup table is actually two triangular 
two-dimensional matrices. To increase the accuracy of computation and economy of memory 
use, the first dimension represents the largest image brightness. The second dimension 
lepiesents to the second largest. Each element of the matrix contains the corresponding 
surface orientation and the smallest image brightness. 

Since the non-linearity gives rise to two solutions for each intensity pair, we have to decide 
which one is the real one. We choose a surface orientation by comparing the distance 
between the actual third image brightness and the element of the matrix. The third 
brightness is always weak and likely to contain errors. It is only used to decide which of the 
two alternatives obtained from the other brightness is the correct one 
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4 Experiment and Discussion 


Our experimental results are shown in Fig'. 10. Fig. 10.1.1 shows three brightness arrays. 
These arrays are input information to the photometric system. No surface normals are 
shown in areas where insufficient information was available in the three images to 
determine them accurately (see Fig. 10.1.2). The choice of light source distribution affects the 
extent of these regions as well as tHe accuracy with which surface normals can be found. 
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Fig. 10.1.1 Three 
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These arrays are 


brightness arrays. Each array corresponds to one of the three lamps, 
input information to the photometric stereo system. 
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Fig. 10.1.2 Direct output from the system. No surface normals are shown in areas where 
there was insufficient information. 
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We d ied two kinds of relaxation methods. The first one is suitable to on-line systems. This 
method reads from the lookup table twice exchanging the second and the third brightness 
values and determines the solution by averaging the two previous solutions. This simple 
scheme can give good results as shown in Fig. 10.1.3. The other method is to find solutions 
which minimize the difference of the actual brightness and the theoretical brightness with a 
constraint derived from surface continuity [83. Theoretical brightness is calculated from 
orientation and reflectance maps. Actually, this method is done iteratively. The assumption 
of surface continuity requires that neighboring points should project to neighboring points 
on the gaussian sphere. Fig. 10.1.4 shows the result obtained using the iterative method. 
You can notice that over a very wide area the method can find surface orientations. This 

method is appropriate for off-line systems. Fig. 10.1.5 is a generated surface from their 
surface orientations. 
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Fig. 10.1.3 Output from the first relaxation 
method. This method would be suitable for a 
hand-eye system where real-time response is 
vital. The system reads from the lookup table 
twice, exchanging the second and the third 
brightness values. The solution is determined 
by averaging the two previous values. This 
simple schema can give good results. 


Fig. 10.1.4 Output from the iterative relaxation 
method This method finds solutions which 
minimize the difference of the actual 
brightness and the theoretical brightness 
using a constraint derived from surface 
continuity. 
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(b) A surface along the x axis. The bold line a surface along the y-axis. 

represents the actual surface and the diagram 
is obtained from the photometric stereo 
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A direct application of our technique is an industrial hand-eye system that picks up an 
object out of a jumble of material (see Fig. 10.2). Although our technique cannot correctly 
determine the surface orientation when there is mutual illumination, it does provide the 
means for detecting this condition, since the three measurements will be inconsistent. In this 
fashion erroneous re.sults are avoided, This is important, since the manipulator might 
otherwise be sent to a position where it would collide with other parts. 
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Fig. 10.2.1 Binary image of a jumble of nuts 
and bolts. 


Fig. 10.2.2 Needle diagram obtained from 
image of nuts and bolts. 



Fig. 10.2.3 The generated surfaces are elevated in order to easily distinguish from the 
surrounding area. 
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Another application of this technique is the inspection of the surface condition of metals. If 
a surface has a crack, stain, or finger print, the image brightness triple yields inconsistent 
values in the area of the blemish (see Fig. 10.3). 



Fig. 10.3.1 A picture of a hook which contains a lot of cracks and stains. 
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Fig. 10.3.2 Needle diagram of the hook. There exists a printed figure 3 in the central area 
of the object. This can be seen as randomness of needles in the area. At the low right side 
the surface is rather dark and the system is unable to determine the surface orientations at 
that area. Note that a relaxation method would fail to identify this central region as 
different from its surroundings 
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This technique may be combined with the Marr-Poggio-Crimson stereo technique now 
being developed [ 9,10 ]. Their technique detects depth cues and works well when the image 
contains many discontinuities. On the other hand our method detects surface orientation 
directly and works well when the object is smooth. The combination can be established if 
the output of one of the two stereo cameras is fed to our system while the two outputs from 
two camera are fed to their system. We feel that the composite system will produce an 
excellent representation of the object just as a people are believed to use both stereo and 
shading information to construct a symbolic image of the visual world. 
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Appendix 

This appendix has two purposes; one is to discover what kind of characteristics, generally 
speaking, are appropriate for a reflectance function, and the other is to answer to these 
questions about the solution of non-linear equations in one space is same as that which is 

f > . 

obtained by solving corresponding equations in another space and being transformed with 
a conformal mapping. 

Proposition 1 

2 

A surface (p s ,q s , R(p s ,q s )) is a regular surface based on the choice of R(p s ,q s ) such that R :U 
-> R is a differentiable one-value function. 



Fig. A.l 


(proof) 

A regular surface satisfies 

1. Its mapping function X is differentiable 

2. X is a [homeomorphism 

3. X satisfies the regularity condition[ 4 ], 

where we can denote X(p s ,q s Mp s ,q s , R(p s ,q s )) as the mapping function from a open set U in 
R 2 to a open set W in R 3 . 

It is obvious that X is differentiable with respect to p 8 and q s , because of the 
differentiability of R(p s ,q s ). Condition 3 requires at least one of the three two-dimensional 
minor jaccobian is always non-zero. Since d(p s ,<\ s )l d(p s ,(\ s ) > s always one, condition 3 is 
satisfied. To each unique (p s .q s ) there exists R(p s ,q s ) which is a one-to-one corresponding to 
<p s ,q s ) because R(p s ,q s ) is one-value differential function. This implies X is one-to-one 
differential mapping. From satisfaction of condition 3 and this one-to-one mapping, inverse 
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theorem guarantees the existence of differential inverse X' 1 . So X is a 
homeomorphism. (qed) 

Now we have known that the surface is a regular surface provided that we choose a single- 
valued differentiable function as a reflectance function; the surface has various 
characteristics as not that dependent on the choice of the parameterization but intrinsic 
ones. The following proposition is rather obvious. 

Proposition 2 The reflectance map of a specular surface can be parameterized with (p n ,q n ) 
such that Eq. A.I, provided that we choose a single-valued differentiable function as a 
reflectance function. 

Ps = 2 p n / (l-p„ 2 -q n z ) 

q s - 2 q n / (l-p n Z -q n 2 ). (A.I) 



(proof) 

A mapping Y:VcR 2 ->UcR 2 such that (p s ,q s ) = Y(p n ,q„)is a differentiable map, where U, V 
are open sets in R z From the theorem of the chain rule Of maps, Z(p (t ,q n Mp„,q n ,R(p n ,q n )) is 
also a differentiable map. (qed) 

For the readers convenience: 

Inverse Function tlieorem[5] 

Let F:UcR"->R n be a differentiable mapping and suppose that at PeU the differential 
dF p :R n ->R n is an isomorphism. Then, there exists a neighborhood V of p in U and a 
neighborhood W of F(p) in R n such that F:v->W has a differentiable inverse F"’:W->V. 
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The Chain Rule for Maps[5] 

Let F:UcR n ->R m and G:VcR m ->R k be differentiable maps, where U and V are open sets 
such that F(U)cV. Then, G°F:U->R k is a differentiable map, and 

d(G°F)p=dGp (p) °dF p , peU. 












